Systems and methods for detecting defective camera arrays and optic arrays

ABSTRACT

Systems and methods for detecting defective camera arrays, optic arrays and/or sensors are described. One embodiment includes capturing image data using a camera array; dividing the captured images into a plurality of corresponding image regions; identifying the presence of localized defects in any of the cameras by evaluating the image regions in the captured images; and detecting a defective camera array using the image processing system when the number of localized defects in a specific set of image regions exceeds a predetermined threshold, where the specific set of image regions is formed by: a common corresponding image region from at least a subset of the captured images; and any additional image region in a given image that contains at least one pixel located within a predetermined maximum parallax shift distance along an epipolar line from a pixel within said common corresponding image region within the given image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The current application is a divisional of application Ser. No.13/931,724 filed Jun. 28, 2013, which claims priority under 35 U.S.C.§119(e) to U.S. Provisional Patent Application Ser. No. 61/665,724,filed Jun. 28, 2012, the disclosures of which are incorporated herein byreference.

FIELD OF THE INVENTION

The present invention generally relates to systems and methods forscreening cameras for defects and more specifically to systems andmethods for screening camera arrays or optic arrays that are used in theconstruction of camera arrays for defects.

BACKGROUND

In response to the constraints placed upon a traditional digital camerabased upon the camera obscura, a new class of cameras that can bereferred to as array cameras has been proposed. Array cameras arecharacterized in that they include an imager array that has multiplearrays of pixels, where each pixel array is intended to define a focalplane, and each focal plane has a separate lens stack. Typically, eachfocal plane includes a plurality of rows of pixels that also forms aplurality of columns of pixels, and each focal plane is contained withina region of the imager that does not contain pixels from another focalplane. An image is typically formed on each focal plane by itsrespective lens stack. In many instances, the array camera isconstructed using an imager array that incorporates multiple focalplanes and an optic array of lens stacks.

SUMMARY OF THE INVENTION

Systems and methods in accordance with embodiments of the inventiondetect defective camera arrays, and/or optic arrays and/or sensors usedin the construction of array camera modules. One embodiment of themethod of the invention includes: capturing image data of a known targetusing the plurality of cameras, where the image data forms a pluralityof images; dividing each of the plurality of images into a plurality ofcorresponding image regions using the image processing system;identifying the presence of at least one localized defect in at leastone of the plurality of the cameras by evaluating the image regions inthe plurality of images in accordance with at least one predeterminedlocalized defect criterion using the image processing system; detectinga defective camera array using the image processing system when thenumber of localized defects in a specific set of image regions exceeds apredetermined threshold. In addition, the specific set of image regionsis formed by: a common corresponding image region from at least a subsetof the plurality of images; and any additional image region in a givenimage that contains at least one pixel located within a predeterminedmaximum parallax shift distance along an epipolar line from a pixelwithin said common corresponding image region within the given image,where the epipolar line is defined by the relative location of thecenter of the camera that captured the given image and a predeterminedviewpoint.

In a further embodiment of the method of the invention, identifying thepresence of at least one localized defect in at least one of theplurality of the cameras by evaluating the image regions in theplurality of images in accordance with at least one predeterminedlocalized defect criterion using the image processing system comprisesidentifying a plurality of defective pixels within an image region thatsatisfies at least one predetermined criterion.

In another embodiment of the method of the invention, the predeterminedcriterion is that the plurality of defective pixels within the imageregion exceeds a predetermined number of defective pixels.

In a still further embodiment of the method of the invention, thepredetermined criterion is that the plurality of defective pixelsincludes a cluster of defective pixels that exceeds a predetermine size.

In still another embodiment of the method of the invention, defectivepixels comprise hot pixels, bright pixels and dark pixels.

In a yet further embodiment of the method of the invention, identifyingthe presence of at least one localized defect in at least one of theplurality of the cameras by evaluating the image regions in theplurality of images in accordance with at least one predeterminedlocalized defect criterion using the image processing system comprises:measuring the Modulation Transfer Function (MTF) within an image region;and determining that the MTF of the image region fails to satisfy apredetermined criterion.

In yet another embodiment of the method of the invention, thepredetermined criterion is that the on-axis MTF at a predeterminedspatial frequency exceeds a first predetermined threshold, the off-axistangential MTF at a predetermined spatial frequency exceeds a secondpredetermined threshold, and the off-axis sagittal MTF at apredetermined spatial frequency exceeds a third predetermined threshold.

In a further embodiment again of the method of the invention, saidplurality of corresponding images portions forms a first plurality ofcorresponding image regions and the method further comprises: dividingeach of the plurality of images into a second plurality of correspondingimage regions using the image processing system, where the number ofimage regions in the first plurality of corresponding image regionsdiffers from the number of image regions in the second plurality ofcorresponding image regions; and identifying the presence of at leastone localized defect in at least one of the plurality of the cameras byevaluating the image regions in the second plurality of images inaccordance with at least one additional predetermined localized defectcriterion using the image processing system.

In another embodiment again of the method of the invention, theplurality of images forms a reference image and a plurality of alternateview images; the specific set of image regions is formed by: a specificimage region from the reference image; the image regions from each ofthe alternate view images that correspond to the specific image regionfrom the reference image; and any additional image region in a givenalternate view image from the plurality of alternate view images thatcontains at least one pixel located within a predetermined maximumparallax shift distance along an epipolar line from a pixel within theimage region of the given alternate view image that corresponds to theselected image region from the reference image, where the epipolar lineis defined by the relative location of the center of the camera thatcaptured the reference image and the center of the camera that capturedthe given alternate view image.

In a further additional embodiment of the method of the invention, theplurality of images forms a plurality of images in each of a pluralityof color channels; and a specific set of image regions is formed byimage regions from the plurality of images within one of the pluralityof color channels.

In another additional embodiment of the method of the invention, theplurality of images forms a reference image and a plurality of alternateview images and said plurality of images from one of the plurality ofcolor channels does not include the reference image; and the specificset of image regions is further formed by: the image regions from eachof the alternate view images within said one of the plurality of colorchannels that correspond to a specific image region from the referenceimage; and any additional image region in a given alternate view imagefrom said one of the plurality of color channels that contains at leastone pixel located within a predetermined maximum parallax shift distancealong an epipolar line from a pixel within the image region of the givenalternate view image that corresponds to the selected image region fromthe reference image, where the epipolar line is defined by the relativelocation of the center of the camera that captured the reference imageand the center of the camera that captured the given alternate viewimage.

A still yet further embodiment of the method of the invention alsoincludes detecting a defective camera array using the image processingsystem when the number of localized defects in a second set of imageregions exceeds a second predetermined threshold, where the second setof image regions is formed by image regions from the plurality of imageswithin a second of the plurality of color channels.

In still yet another embodiment of the method of the invention, saidpredetermined criterion used with respect to said specific set of imageregions from said one of the plurality of color channels is differentfrom said second predetermined criterion used with respect to saidsecond set of image regions from said second of the plurality of colorchannels.

A still further embodiment again of the method of the invention includesdividing the image field of each of the plurality of lens stacks into aplurality of corresponding regions when using an optical testinstrument; measuring the Modulation Transfer Function (MTF) of a knowntarget using the optical test instrument in each of the regions;identifying the presence of at least one localized defect in at leastone of the plurality of the lens stacks by evaluating the MTFmeasurements of the regions in the plurality of lens stacks inaccordance with at least one predetermined localized defect criterionusing the optical test instrument; detecting a defective optic arrayusing the image processing system when the number of localized defectsin a specific set of regions exceeds a predetermined threshold. Inaddition, the specific set of regions is formed by: a commoncorresponding region from at least a subset of the plurality of lensstacks; and any additional region in a given lens stack that forms animage within a predetermined maximum parallax shift distance along anepipolar line from said common corresponding region within the givenlens stack, where the epipolar line is defined by the relative locationof the center of the given lens stack and a predetermined viewpoint.

Another further embodiment of the method of the invention includescapturing image data using a camera array comprising a plurality ofcameras, where at least one of the plurality of cameras includes a knownlocalized defect impacting image data captured by the camera;disregarding image data within a region of an image captured by the atleast one of the plurality of cameras that includes a known localizeddefect using a processor configured by a super-resolution imageprocessing application, where the discarded image data is from a regionof the camera that is known to include said known localized defect; andsynthesizing a super-resolution image from the remaining image datacaptured by the cameras in the camera array using a super-resolutionprocess performed by the processor configured using the super-resolutionimage processing application.

In still another further embodiment of the method of the invention, thecamera array comprises at least one camera known to include at least onedefective pixel, and the method further comprises disregarding imagedata captured by the pixels in the at least one camera that are known tobe defective.

Another embodiment of the invention includes: an array camera modulecomprising a plurality of cameras formed by an imager array comprising aplurality of and an optic array comprising a plurality of lens stacks,where at least one of the plurality of cameras formed by the imagerarray and optic array includes a known localized defect impacting imagedata captured by the camera; a processor; and memory containing asuper-resolution image processing application and defect dataidentifying said at least one of the plurality of cameras that includesa known localized defect and a region of the camera that contains theknown localized defect. In addition, the super-resolution processingapplication configures the processor to: capture image data using thearray camera module; with respect to each of said at least one of theplurality of cameras that includes a known localized defect,disregarding image data within at least one region identified by thedefect data; and synthesizing a super-resolution image from theremaining image data.

In still another embodiment of the invention, memory further comprisesdefect data identifying at least one defective pixel within the imagerarray and the super-resolution processing application configures theprocessor to also disregard image data captured by the at least onepixel identified as defective by said defect data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 conceptually illustrates a camera array implemented in the formof an array camera.

FIG. 2 conceptually illustrates an array camera module constructed froman optic array and an imager array.

FIG. 3 illustrates the circuitry in an image array that can be utilizedin the construction of an array camera module.

FIG. 4 illustrates circuitry utilized in the independent control andread out of pixel sub-arrays that form focal planes on an imager arraythat can be utilized in the construction of an array camera module.

FIGS. 5A-5E conceptually illustrate a process for determining whether acamera array is capable of synthesizing an image having acceptable imagequality from image data captured by the cameras in the camera array inaccordance with embodiments of the invention.

FIG. 6 is a process for determining whether a camera array is defectivedue to the presence of localized defects in predetermined parallaxuncertainty zones in accordance with an embodiment of the invention.

FIG. 7 is a process for determining whether a region of a camera isdefective based upon the presence of defective pixels within the regionof the camera in accordance with an embodiment of the invention.

FIG. 8 is a process for determining whether a region of a camera isdefective based upon measurements of the MTF of the camera in the regionin accordance with an embodiment of the invention.

FIG. 9 is a process for selecting a reference camera within a cameraarray based upon the reference camera being free from defective regionsin accordance with an embodiment of the invention.

FIG. 10 is a process for determining whether an optic array is defectivebased upon the presence of localized regions of MTF that do not pass anacceptance criterion in predetermined parallax uncertainty zones inaccordance with an embodiment of the invention.

FIG. 11 is a process for synthesizing a super-resolution image fromimage data captured by a camera array including defects usinginformation concerning the location of the defective regions todisregard captured image data that are impacted by the defects inaccordance with an embodiment of the invention.

DETAILED DESCRIPTION OF THE DRAWINGS

Turning now to the drawings, systems and methods for detecting defectivecamera arrays, and/or optic arrays and/or sensors used in theconstruction of array camera modules in accordance with embodiments ofthe invention are illustrated. A variety of defects can arise during themanufacture of a conventional digital camera that captures images usinga single aperture including (but not limited to) defects in the cameraoptics including defects that result in unacceptable Modulation TransferFunction (MTF) performance, defective pixels in the camera's sensor,and/or defects in the assembly of the optics and sensor to form thecamera. With respect to the discussion below, the term defect is used torefer to any aspect of a camera (including the sensor, optics, and/orassembly) or optic array that negatively impacts the image formed and/orimage data captured by the camera. Even when a defect is localized, thedefect can render the camera unsuitable for use, as the localized defectwill result in unsatisfactory image quality in the impacted region ofevery image captured by the camera. As is discussed below, system andmethods in accordance with embodiments of the invention utilizeknowledge of the image processing used to synthesize images from imagescaptured by camera arrays to determine whether localized defects inspecific cameras in an array can be tolerated. In this way, yield can beimproved in the manufacture of camera arrays by utilizing camera arraysthat contain defects that will not impact the performance of the cameraarray.

A variety of camera arrays and processes for manufacturing camera arraysare disclosed in U.S. patent application Ser. No. 12/935,504, entitled“Capturing and Processing of Images using Monolithic Camera Array withHeterogeneous Images”, filed May 20, 2009, the disclosure of which isincorporated by reference herein in its entirety. Multiple images of ascene can be captured by a camera array and utilized to synthesize ahigher (super-resolution) image of the scene. Fusion andsuper-resolution processes that can be utilized to generatesuper-resolution images using images captured by a camera array aredisclosed in U.S. patent application Ser. No. 12/967,807, entitled“System and Methods for Synthesizing High Resolution Images UsingSuper-Resolution Processes”, filed Dec. 14, 2010, the disclosure ofwhich is incorporated herein by reference in its entirety.

A portion of an image synthesized using a super-resolution processtypically includes image data from multiple images captured by a cameraarray. In many instances, the complete set of images captured by acamera array is not required to achieve acceptable image quality in aregion of a synthesized image. Manufacture of camera arrays results inmany of the same defects that are experienced during the manufacture ofconventional cameras. One approach to determining whether a camera arrayis defective is to identify cameras within the camera array that containdefects and to identify the entire camera array as defective when apredetermined threshold number of defective cameras is exceeded. Forarray cameras that include sets of cameras that form different colorchannels, such as those disclosed in U.S. patent application Ser. No.12/935,504, the number of defective cameras in each color channel can beevaluated with respect to separate predetermined thresholds in order todetermine whether the camera array as a whole is defective. For colorchannels that include fewer cameras, a smaller number of defectivecameras may be tolerated. Although rejecting camera arrays as defectivebased on the number of cameras containing defects is effective, theprocess may reject camera arrays that could still be utilized tosynthesize images of acceptable image quality (despite the presence of apredetermined number of localized defects within a color channel).Increased manufacturing yield can be achieved by identifying theportions of the images captured by the camera array that are impacted bydefects of some cameras and evaluating whether sufficient reliable imagedata remains for that region from all the remaining cameras tosynthesize an image. If sufficient reliable image data remains tosynthesize an image, then the camera array can be utilized irrespectiveof the total number of cameras impacted by localized defects.

In several embodiments, a determination that a camera array is defectiveis made by dividing each of the images of a scene (typically a knowntarget) captured by the cameras in the camera array into correspondingregions and determining which of the regions contain pixels that containimage data which is likely to be fused during image processing to formregions of a synthesized image. In many embodiments, the imageprocessing involves performing a super-resolution process involvingparallax detection and correction to synthesize a super-resolved image.In other embodiments, any of a variety of image processing techniquescan be utilized including (but not limited to processes that synthesize)stereo-pairs of super-resolution images, video sequences synthesizedusing a subset of cameras in the camera array, and/or high frame ratevideo sequences where different frames of video are synthesized usingimage data captured by different subsets of the camera array. In theevent that a predetermined number of the regions that are likely to befused to form a region of a synthesized image are impacted by localizeddefects, then the camera array can be determined to be defective. In thecontext of a super-resolution process, regions that are likely to befused to form a specific region of a synthesized image can be identifiedusing the maximum parallax shifts that are likely to be observed betweenimages captured by the cameras in the camera array. In severalembodiments, one of the cameras in a camera array is selected as areference camera and the remaining cameras are considered alternate viewcameras. In certain embodiments, the reference camera is selected inaccordance with criteria including that the reference camera does notinclude any defects. When a region of an image captured by the referencecamera is considered, the maximum parallax shifts along epipolar linesof the pixels in the region define so called “parallax uncertaintyzones” within each of the alternate view images. A determinationconcerning whether a camera array is defective can be made by countingthe number of defects impacting pixels within the parallax uncertaintyzones associated with each region within the image captured by thereference camera. Where the cameras in a camera array form multiplecolor channels, separate criteria based upon parallax shifts can beapplied to evaluate the impact of the localized defects present in thecameras of each color channel.

As indicated above, a variety of defects can occur during componentmanufacture and assembly of a camera or camera array. In severalembodiments, the process of evaluating whether a camera array isdefective can involve evaluation the cameras in the camera array forseveral different types of defects including (but not limited to)defects in camera optics, defects in the pixels of camera sensors anddefects in the assembly of the camera optics and sensors. In a number ofembodiments, the size of the regions of the images considered whenevaluating the impact of specific types of localized defects can differ.In many embodiments, larger regions are considered when evaluating thecamera's optics in a given region of an image captured by the camerathan the regions considered when evaluating the impact of defectivepixels in the camera's sensor. In general, the smaller the regionsconsidered (i.e. the larger the number of regions considered) during thedefect detection process, the higher the anticipated yield up to a pointat which the process is: identifying all camera arrays in which thedefects present in the camera array can be tolerated by thesuper-resolution processing; and rejecting all camera arrays asdefective where the defects result in insufficient reliable image datafor reliably performing super-resolution processing.

Evaluating the likely impact of localized defects based upon anticipatedparallax shifts can improve overall manufacturing yields, because cameraarrays are not rejected as defective simply based upon a predeterminednumber of cameras in the camera array containing defects. Similartechniques can be utilized to evaluate optic arrays utilized in theconstruction of array camera modules (similar to the array cameramodules discussed in U.S. patent application Ser. No. 12/935,504). Inmany embodiments, a Modulation Transfer Function (MTF) measurement canbe made with respect to different regions of the images formed by eachlens stack in an optic array. MTF is generally the most relevantmeasurement of optical performance, and is generally taken as anobjective measurement of the ability of an optical system to transfervarious levels of detail (or spatial frequency) from an object to animage. The MTF is measured in terms of contrast (degrees of gray), or ofmodulation, produced from a perfect source of that detail level (thus itis the ratio of contrast between the object and the image). The amountof detail in an image is given by the resolution of the optical system,and is customarily specified as spatial frequency in line pairs permillimeter (lp/mm). A line pair is one cycle of a light bar and dark barof equal width and has a contrast of unity. Contrast can be defined asthe ratio of the difference in maximum intensity (I_(max)) and minimumintensity (I_(min)) over the sum of I_(max) and I_(min), where I_(max)is the maximum intensity produced by an image (white) and I_(min) is theminimum intensity (black). The MTF then is the plot of contrast,measured in percent, against spatial frequency measured in lp/mm. Theimpact of errors in the lens such as (but not limited to) centeringerrors, form errors, and/or thickness errors that negatively impact MTFto a point at which the region of a lens stack is considered to containa defect (i.e. MTF measurements that fail to satisfy one or morepredetermined criterion) can be evaluated based upon anticipatedparallax shifts during super-resolution processing. In this way,manufacturing yield can be increased by considering the regions ofimages impacted by defects as opposed to simply the number of defects inan optic array. Systems and methods for detecting defective cameraarrays, and/or optic arrays, and techniques for synthesizingsuper-resolution images from images captured by array cameras containinglocalized defects in accordance with embodiments of the invention arediscussed further below.

Camera Arrays

While much of the discussion that follows refers to systems and methodsfor screening for defective camera arrays, it is worthwhile initiallyreviewing the construction of camera arrays, the defects that can occurin the construction of camera arrays, and the manner in whichinformation concerning localized defects can be utilized whensynthesizing super-resolution images to avoid corruption of thesynthesized super-resolution image by pixels impacted by the localizeddefects. Camera arrays can be implemented in a variety of ways including(but not limited to) as a set of discrete cameras, or as an arraycamera. Array cameras typically can include an array camera module and aprocessor.

An array camera that is configured to synthesize super-resolution imagesin a manner that involves disregarding image data impacted by localizeddefects in the cameras in the array camera in accordance with anembodiment of the invention is illustrated in FIG. 1. The array camera100 includes an array camera module 102 including an array of individualcameras 104 where an array of individual cameras refers to a pluralityof cameras in a particular arrangement, such as (but not limited to) thesquare arrangement utilized in the illustrated embodiment. The arraycamera module 102 is connected to the processor 106 and the processor106 is connected to a memory 108. In a number of embodiments, the memorycontains a super-resolution image processing application that isconfigured to synthesize a super-resolution image using image datacaptured by the camera module 102 using a process such as (but notlimited to) one of the processes outlined in U.S. patent applicationSer. No. 12/967,807. In several embodiments, the memory 108 containsinformation concerning image data captured by the camera module 102 thatis unreliable due to localized defects in the cameras within individualcameras 104 within the camera module 108. The information can be in theform of regions of images that can be disregarded and/or individualpixels or clusters of pixels that can be disregarded. Thesuper-resolution image processing application can utilize theinformation concerning the image data that is unreliable in the capturedimages to disregard the unreliable image data when performingsuper-resolution processing.

Although a specific array camera is illustrated in FIG. 1, any of avariety of different array camera configurations can be utilized inaccordance with many different embodiments of the invention.Furthermore, the basic configuration shown in FIG. 1 can also beutilized in an image processing system that can be utilized to detectthe presence of localized defects in a camera module and to determinewhether the localized defects result in the overall camera module beingdefective for the purposes of synthesizing super-resolution images usingthe image data captured by the individual cameras within the cameramodule.

Array Camera Modules

The defects that can be present in array cameras typically arise fromthe manner in which array camera modules are constructed. Array cameramodules, such as the array camera modules discussed above with respectto FIG. 1, can be constructed from an imager array and an optic array inthe manner illustrated in FIG. 2. The camera module 200 includes animager array 230 including an array of focal planes 240 along with acorresponding optic array 210 including an array of lens stacks 220.Within the array of lens stacks, each lens stack 220 creates an opticalchannel that forms an image of the scene on an array of light sensitivepixels within a corresponding focal plane 240. Each pairing of a lensstack 220 and focal plane 240 forms a single camera 104 within the arraycamera module. Each pixel within a focal plane 240 of a camera 104generates image data that can be sent from the camera 104 to theprocessor 108. In many embodiments, the lens stack within each opticalchannel is configured so that pixels of each focal plane 240 sample thesame object space or region within the scene. In several embodiments,the lens stacks are configured so that the pixels that sample the sameobject space do so with sub-pixel offsets to provide sampling diversitythat can be utilized to recover increased resolution through the use ofsuper-resolution processes.

In several embodiments, color filters in individual cameras can be usedto form multiple color channels within the array camera module. In thisway, cameras can be used to capture data with respect to differentcolors, or a specific portion of the spectrum. In contrast to applyingcolor filters to the pixels of the camera, color filters in manyembodiments of the invention can be included in the lens stack. Forexample, a green color camera can include a lens stack with a greenlight filter that allows green light to pass through the opticalchannel. In many embodiments, the pixels in each focal plane are thesame and the light information captured by the pixels is differentiatedby the color filters in the corresponding lens stack for each filterplane. Although a specific construction of a camera module with an opticarray including color filters in the lens stacks is described above,camera modules can be implemented in a variety of ways including (butnot limited to) by applying color filters to the pixels of the focalplanes of the camera module similar to the manner in which color filtersare applied to the pixels of a camera that uses a conventional Bayercolor filter pattern. In several embodiments, at least one of thecameras in the camera module can include uniform color filters appliedto the pixels in its focal plane. In many embodiments, a Bayer filterpattern is applied to the pixels of one of the cameras in a cameramodule. In a number of embodiments, camera modules are constructed inwhich color filters are utilized in both the lens stacks and on thepixels of the imager array.

The defects that can be present in a camera module include (but are notlimited to) defective pixels, a lens stack including one or more lenssurfaces that deviate from the relevant prescriptions for the surfaces,and defects associated with the manner in which the sensor and the opticarray are combined to form the camera module. The types of defectivepixels that may be present can include (but are not limited to) hotpixels (pixels that generate a signal above a predetermined mean darksignal when the sensor array is not illuminated), bright pixels (pixelsthat produce values that exceed a predetermined threshold above thevalues produced by neighboring pixels under similar illuminationconditions), and dark pixels (pixels that produce values lower than apredetermined threshold below the values produced by neighboring pixelsunder similar illumination conditions). The specific types of pixeldefects that can be detected in accordance with embodiments of theinvention typically depend upon the requirements of a specificapplication. As noted above, a variety of characteristics of the opticsof a camera can result in sufficient deterioration to be considereddefects in accordance with embodiments of the invention. In manyembodiments, defects in a region of a lens can be detected by measuringwhether one or both of the tangential and/or sagittal MTF components(sometimes referred to as the horizontal and vertical components) failto exceed one or more predefined thresholds. Additional defects that canbe detected include (but are not limited to) blemishes in the opticsand/or that result from assembly. Although specific categories oflocalized defect are described above, processes in accordance withembodiments of the invention can evaluate the impact of any of a varietyof defects that are localized to a region of a camera within a cameraarray on the performance of the camera array using informationconcerning the manner in which images will be synthesized from the imagedata captured by the camera array.

Although specific array camera modules and defects that can occur duringthe manufacture of array camera modules are discussed above, manydifferent array camera modules can be constructed and systems andmethods in accordance with embodiments of the invention can detect thepresence of the types of defects that typically arise in theconstruction of a specific type of array camera module. In order toprovide some additional background concerning the operation of imagerarrays and the manner in which the imager arrays capture image data foruse in super-resolution processing, imager arrays that can be utilizedin the construction of array camera modules in accordance withembodiments of the invention are discussed further below.

Imager Arrays

An imager array that can be utilized in the construction of an arraycamera module and in which the image capture settings of a plurality offocal planes can be independently configured is illustrated in FIG. 3.The imager array 300 includes a focal plane array core 302 that includesan array of focal planes 304 and all analog signal processing, pixellevel control logic, signaling, and analog-to-digital conversion (ADC)circuitry. The imager array also includes focal plane timing and controlcircuitry 306 that is responsible for controlling the capture of imageinformation using the pixels. In a number of embodiments, the focalplane timing and control circuitry utilizes reset and read-out signalsto control the integration time of the pixels. In other embodiments, anyof a variety of techniques can be utilized to control integration timeof pixels and/or to capture image information using pixels. In manyembodiments, the focal plane timing and control circuitry 306 providesflexibility of image information capture control, which enables featuresincluding (but not limited to) high dynamic range imaging, high speedvideo, and electronic image stabilization. In various embodiments, theimager array includes power management and bias generation circuitry308. The power management and bias generation circuitry 308 providescurrent and voltage references to analog circuitry such as the referencevoltages against which an ADC would measure the signal to be convertedagainst. In many embodiments, the power management and bias circuitryalso includes logic that turns off the current/voltage references tocertain circuits when they are not in use for power saving reasons. Inseveral embodiments, the imager array includes dark current and fixedpattern (FPN) correction circuitry 310 that increases the consistency ofthe black level of the image data captured by the imager array and canreduce the appearance of row temporal noise and column fixed patternnoise. In several embodiments, each focal plane includes referencepixels for the purpose of calibrating the dark current and FPN of thefocal plane and the control circuitry can keep the reference pixelsactive when the rest of the pixels of the focal plane are powered downin order to increase the speed with which the imager array can bepowered up by reducing the need for calibration of dark current and FPN.

In many embodiments, a single self-contained chip imager array includesfocal plane framing circuitry 312 that packages the data captured fromthe focal planes into a container file and can prepare the capturedimage data for transmission. In several embodiments, the focal planeframing circuitry includes information identifying the focal planeand/or group of pixels from which the captured image data originated. Ina number of embodiments, the imager array also includes an interface fortransmission of captured image data to external devices. In theillustrated embodiment, the interface is a MIPI CSI 2 output interface(as specified by the non-profit MIPI Alliance, Inc.) supporting fourlanes that can support read-out of video at 30 fps from the imager arrayand incorporating data output interface circuitry 318, interface controlcircuitry 316 and interface input circuitry 314. Typically, thebandwidth of each lane is optimized for the total number of pixels inthe imager array and the desired frame rate. The use of variousinterfaces including the MIPI CSI 2 interface to transmit image datacaptured by an array of imagers within an imager array to an externaldevice in accordance with embodiments of the invention is described inU.S. Pat. No. 8,305,456, entitled “Systems and Methods for TransmittingArray Camera Data”, issued Nov. 6, 2012, the disclosure of which isincorporated by reference herein in its entirety.

Although specific components of an imager array architecture arediscussed above with respect to FIG. 3, any of a variety of imagerarrays can be constructed in accordance with embodiments of theinvention that enable the capture of images of a scene at a plurality offocal planes in accordance with embodiments of the invention.Independent focal plane control that can be included in imager arrays inaccordance with embodiments of the invention are discussed furtherbelow.

Independent Focal Plane Control

Imager arrays in accordance with embodiments of the invention caninclude an array of focal planes that can independently be controlled.In this way, the image capture settings for each focal plane in animager array can be configured differently. An imager array includingindependent control of image capture settings and independent control ofpixel readout in an array of focal planes in accordance with anembodiment of the invention is illustrated in FIG. 4. The imager array400 includes a plurality of focal planes or pixel sub-arrays 402.Control circuitry 403, 404 provides independent control of the exposuretiming and amplification gain applied to the individual pixels withineach focal plane. Each focal plane 402 includes independent row timingcircuitry 406, 408, and independent column readout circuitry 410, 412.In operation, the control circuitry 403, 404 determines the imagecapture settings of the pixels in each of the active focal planes 402.The row timing circuitry 406, 408 and the column readout circuitry 410,412 are responsible for reading out image data from each of the pixelsin the active focal planes. The image data read from the focal planes isthen formatted for output using an output and control interface 416.

Although specific imager array configurations are discussed above withreference to FIG. 4, any of a variety of imager array configurationsincluding independent and/or related focal plane control can be utilizedin accordance with embodiments of the invention including those outlinedin U.S. patent application Ser. No. 13/106,797, entitled “Architecturesfor Imager Arrays and Array Cameras”, filed May 12, 2011, the disclosureof which is incorporated by reference herein in its entirety. As isdiscussed further below, the image data captured by an imager array canbe utilized to detect localized defects in the cameras formed by anarray camera module and to evaluate whether the defects will ultimatelyrender the entire array camera module defective.

Evaluating Defects in Camera Arrays

Camera arrays can capture information in multiple color channels orspectral cameras, where specific cameras are configured to only captureimage data within a single color channel or spectral band. A 4×4 cameraarray that is configured to capture red, green, and blue image data isconceptually illustrated in FIG. 5A. As noted above, super-resolutionprocesses including (but not limited to) the process disclosed in U.S.patent application Ser. No. 12/967,807 can be utilized to take imagedata captured by each of the cameras in the camera array and synthesizea super-resolution image. The process of synthesizing a super-resolutionimage from images captured by multiple cameras having differentviewpoints involves identifying pixel shifts that can be applied to theimage data to shift all of the captured image data to a singleviewpoint. Referring to the camera array illustrated in FIG. 5A, areference camera 500 can be designated and all of the remaining camerascan be considered alternate view cameras. One approach to determiningthe appropriate shifts to apply to the pixels of the images captured bythe alternate view cameras to shift the pixels into the viewpoint of thereference camera is to determine the distance to objects within thescene captured by the reference camera. These distances can then be usedto determine the anticipated parallax shifts in the alternate viewimages, which can then be corrected. The parallax shifts in thealternate view images will typically occur along epipolar lines, whichare determined based upon the relative locations of the centers of thereference camera and the alternate view camera. Processes for detectingand correcting for parallax shifts are disclosed in U.S. ProvisionalPatent Application Ser. No. 61/780,906, entitled “Systems and Methodsfor Parallax Detection and Correction in Images Captured Using ArrayCameras”, filed Mar. 13, 2013, the disclosure of which is incorporatedby reference herein in its entirety. As is discussed in U.S. ProvisionalPatent Application Ser. No. 61/780,906, disparity searches performedwhen conducting parallax detection and correction can be bounded basedupon a maximum observed parallax shift. As is discussed further belowwith reference to FIGS. 5B-5E, an appreciation of these bounds can beutilized to determine whether localized defects in the cameras of acamera array will impact the image quality of images synthesized usingimages captured by the camera array.

Using Bounds on Parallax Shifts to Evaluate Impact of Localized Defects

Systems and methods for screening camera arrays for defects inaccordance with many embodiments of the invention attempt to evaluatewhether the image data captured by a camera array includes sufficientreliable image data to reliably synthesize a super-resolution image. Inseveral embodiments, the sufficiency of the captured image data isdetermined by considering the super-resolution image as a set of regionssynthesized from image data captured by pixels in regions in each of theimages captured by the camera array. While the locations of the regionsin the images correspond with the locations of the regions in thesuper-resolution image, it should be noted that the effects of parallaxcan mean that a region of a super-resolution image can be synthesizedfrom image data captured from more than just the corresponding regionsof the images captured by the camera array. The process of synthesizinga region of the super-resolution image involves shifting all of theimage data captured by the cameras in the camera array to the viewpointfrom which the super-resolution image is synthesized, which can includeshifting image data captured by pixels from multiple regions of acamera. Although much of the discussion that follows assumes that thesuper-resolution image is synthesized from the viewpoint of a referencecamera, super-resolution images can also be synthesized from virtualviewpoints. In which case, parallax corrections are applied to all ofthe image data.

The reliability with which the region of the super-resolution image canbe synthesized based upon captured image data can be evaluated byidentifying pixels in the image data that could be utilized tosynthesize the super-resolution image and which may be impacted by alocalized defect. As noted above, the parallax shifts that are likely tobe observed in captured image data are typically bounded. Therefore,these maximum parallax shift bounds can be utilized to identify pixelsin image data captured by specific cameras within an array that could beutilized to synthesize a region of a super-resolution image dependingupon the nature of a scene. The specific pixels that will be utilized tosynthesize a region of a super-resolution image will typically dependupon the distance(s) to objects within the scene that are visible withinthe synthesized region of the super-resolution image. Regions of theimages captured by specific cameras within a camera array that containpixels that could be utilized to synthesize a region of asuper-resolution image (identified based upon the parallax shift bounds)can be referred to as parallax uncertainty zones with respect to theregion of the super-resolution image. These parallax uncertainty zonescontain the pixels that could be utilized to synthesize the associatedregion of the super-resolution image under all possible imagingconditions (i.e. across all possible object distances). By identifyingthe number of localized defects (if any) that impact pixels containedwithin the parallax uncertainty zones, systems and methods in accordancewith embodiments of the invention can identify the amount of image datathat must be disregarded during the synthesis of the region of thesuper-resolution image. If the amount of image data that must bedisregarded (i.e. the number of localized defects impacting pixelscontained within the parallax uncertainty zones) exceeds a predeterminedamount, then the camera array can be determined to be defective for thepurpose of synthesizing super-resolution images. Although much of thediscussion that follows focuses on synthesizing super-resolution images,similar processes can be utilized to evaluate whether sufficientreliable image data is available to synthesize other types of image(s)such as (but not limited to) stereo-pairs of super-resolution images,and/or video sequences synthesized using a subset of cameras in thecamera array.

In order to provide a concrete example of the processes outlined above,a process for determining whether localized defects in the cameras ofthe 4×4 camera array illustrated in FIG. 5A render the camera arraydefective for the purpose of synthesizing super-resolution images fromthe viewpoint of the reference camera 500 in accordance with anembodiment of the invention is conceptually illustrated in FIGS. 5B-5E.The 4×4 camera array illustrated in FIG. 5A includes cameras thatcapture red, green, and blue image data. In evaluating the camera array,each color channel can be considered separately. Referring first to FIG.5B, the sufficiency of the reliable image data captured by cameraswithin the green color channel is considered with respect to a region ofthe super-resolution image defined by dividing the super-resolutionimage into a 3×3 grid and dividing each of the images captured by thecamera array into corresponding 3×3 grids. Although regions definedusing 3×3 grids are utilized to illustrate the process shown in FIGS.5B-5E, the number of regions can be selected based upon the requirementsof specific applications and, as is discussed further below, the size ofthe regions can differ when considering different types of defects thatmay be present within a given camera array. Due to the fact that thesuper-resolution image is synthesized from the viewpoint of thereference camera 500, the region of the super-resolution image that isbeing considered corresponds to a region 502 of the reference camera(i.e. the anticipated parallax shifts to shift image data captured bypixels of the reference camera into the viewpoint of the synthesizedsuper-resolution image is zero). In the illustrated embodiment, epipolarlines 504 and maximum parallax shifts are utilized to identify regionswithin the alternate view green cameras that contain pixels that couldpotentially capture image data that could be utilized to synthesize theregion of the super-resolution image under all possible imagingconditions. In the illustrated embodiment, a maximum parallax shift isassumed that is approximately equal to the relevant dimension of one ofthe regions (i.e. somewhere between the length of the diagonal and thelength of an edge of the region depending on the location of the camerawithin the array). In actual applications, the maximum parallax shiftthat is observed typically depends upon the location of an alternateview camera relative to the reference camera. In certain embodiments,different maximum parallax shifts are utilized based upon cameralocation. In other embodiments, the same maximum parallax shift isutilized irrespective of the camera location to simplify analysis. Thespecific parallax shift(s) that are assumed typically depend upon thespacing and focal length of the cameras in a specific camera array.

The identified regions within the alternate view green cameras thatcontain at least one pixel that could potentially capture image dataused to synthesize a given region of the super-resolution image definethe parallax uncertainty zones 506 for the given region of thesuper-resolution image. In FIGS. 5B-5E, parallax uncertainty zones areillustrated as shaded regions within each camera. Once the parallaxuncertainty zones are identified, the process of determining whether thecamera array captures sufficient reliable image data to reliablysynthesize super-resolution images under all imaging conditions involvessimply counting the number of defects that impact pixels within theparallax uncertainty zones. When the counted number of defects withinany of the parallax uncertainty zone exceeds a predetermined number,then the camera array is considered defective. In a number ofembodiments, less than three localized defects impacting regions withinthe parallax uncertainty zones can be tolerated. Referring again to FIG.5B, localized defects are indicated using the symbol “X”. As can bereadily appreciated, the presence of three localized defects (X) withinthe uncertainty zones of the region of the super-resolution image beingevaluated would result in the illustrated camera array being considereddefective.

The ability to define parallax uncertainty zones in a predeterminedmanner can simplify processes for detecting defective camera arrays inaccordance with embodiments of the invention during the manufacturing ofcamera arrays. Processes for determining whether camera arrays aredefective can simply involve determining regions of the cameras thatcontain localized defects and then using lookup tables to identify theparallax uncertainty zones to consider when evaluating whether thelocalized defects render the overall camera array defective for thepurpose of synthesizing a desired type of image(s).

In many embodiments of the invention, the process of evaluating whethera camera array is defective involves evaluating whether regions withinparallax uncertainty zones contain localized defects. The regions of acamera that are considered part of a parallax uncertainty zone for agiven region of a super-resolution image are regions that contain atleast one pixel that could potentially capture image data that could beutilized to synthesize the given region of the super-resolution imageunder all possible imaging conditions. It is also worth noting that aregion that is part of a parallax uncertainty zone can also includepixels that capture image data that will not be utilized by thesuper-resolution process to synthesize the given region of thesuper-resolution image under any imaging conditions (i.e. pixels thatshift along epipolar lines to different regions of the super-resolutionimage). For example, region 508 contains such pixels. In the event thatthe localized defect impacting region 508 does not impact pixels thatcould potentially capture image data used in the synthesis of the givenregion of the super-resolution image, then the camera array couldtheoretically still be used to synthesize super-resolution images ofacceptable image quality despite failing to the criterion outlinedabove. Accordingly, yield can be further increased by reducing the sizeof the regions (i.e. using more than a 3×3 grid, e.g. a 6×8 grid and/orany other grid appropriate to the requirements of the invention). As isdiscussed further below, the size of the regions considered can dependupon by the specific type of defect being detected. For example,defective pixels can be identified individually and so very smallregions can be considered when evaluating the impact of defectivepixels. By contrast, MTF calculations typically require image datacaptured by a larger number of pixels. Therefore, larger regions may beutilized when evaluating the impact of defects in the lens stacks ofoptic array of an array camera module. In addition, the size, number andlocation of the regions for testing the optic array only can be alreadydefined by the setup of the MTF testing equipment, e.g. if opticaltesting instrument use 9 reticles and corresponding cameras in thetester (on-axis, 4 at intermediate field heights on H and V axes of“image” and 4 in the corners). Accordingly, a single screening processcan utilize different sized regions when evaluating the impact ofdifferent types of defects as part of the process of determining whethera camera array is sufficiently reliable to be utilized in synthesizing adesired type of image.

The manner in which parallax uncertainty zones are defined with respectto various regions of a super-resolution image can be furtherappreciated with reference to FIGS. 5C-5E. With specific regard to FIGS.5C and 5D, different regions of the super-resolution image are selectedcorresponding to region 510 and region 520 in the reference camera(shown in FIG. 5C and FIG. 5D respectively). Epipolar lines 512, 522 andmaximum parallax shift bounds are utilized to identify parallaxuncertainty regions 514, 524 and the number of localized defectsimpacting pixels within the parallax uncertainty zones 514, 524 can thenbe determined. When the number of localized defects impacting pixelswithin the parallax uncertainty zones exceeds a predetermined thresholdnumber, then the camera array can be considered defective for thepurpose of synthesizing super-resolution images.

The reference camera 500 in the camera array illustrated in FIG. 5A is acamera that captures image data within a green color channel. Theprocess of synthesizing a super-resolution image can also involveshifting image data captured by cameras within other color channels tothe viewpoint of the reference camera. In several embodiments, theprocess of evaluating whether a camera array can reliably synthesize asuper-resolution image involves evaluating whether defects in camerasthat are part of a color channel that does not contain the referencecamera are likely to result in unacceptable image quality in asuper-resolution image synthesized using image data captured by thecamera array. The process for evaluating the likely impact of localizeddefects in cameras that are part of a color channel that does notcontain the reference camera is similar to the process outlined above.Epipolar lines and maximum parallax shift bounds are utilized toidentify regions within the alternate view cameras within the colorchannel that constitute parallax uncertainty zones for a specific regionof a synthesized super-resolution image. In many instances, the numberof cameras within the camera array used to capture image data indifferent color channels may vary. Therefore, a different threshold maybe utilized to determine whether an array camera is defective in eachcolor channel.

A process for evaluating whether the camera array illustrated in FIG. 5Ais defective for the purpose of synthesizing a full-colorsuper-resolution image due to localized defects present in cameras thatare part of a blue color channel in accordance with embodiments of theinvention is conceptually illustrated in FIG. 5E. The process involvesselecting a region of the super-resolution image that corresponds to aregion 530 of the reference camera 500. Although the region 530corresponding to the selected region of the super-resolution image isshown in FIG. 5E, the reference camera does not capture image data inthe blue color channel and so the reference camera is not considered forthe purposes of evaluating the cameras in the blue color channel. Theregion 330 is simply shown for the purpose of illustrating the manner inwhich the parallax uncertainty regions within the cameras that are partof the blue channel are determined. Epipolar lines and maximum parallaxshift bounds are utilized to identify parallax uncertainty regions inthe cameras within the blue channel with respect to the selected regionof the super-resolution image. A determination can be made as to thenumber of localized defects (of any type) that impact regions within theparallax uncertainty zones of the cameras within the blue color channel.Where the number of regions within the parallax uncertainty zonesimpacted by localized defects exceeds a predetermined threshold, thenthe camera array can be determined to be defective for the purpose ofsynthesizing full-color super-resolution images. With specific regard tothe 4×4 camera array illustrated in FIG. 5A, the number of defects thatcan be tolerated in the parallax uncertainty zones of a specific regionof a super-resolution image before the camera array is considereddefective is a single defect. In other embodiments, the number oflocalized defects that are tolerated within the parallax uncertaintyzones can be determined based upon the requirements of a specificapplication.

Although the processes discussed above with respect to FIG. 5E arediscussed in the context of the cameras that form the blue color channelin the camera array shown in FIG. 5A, similar processes can be appliedto determine whether defects in the cameras that form the red colorchannel compromise the reliability with which super-resolution imagescan be synthesized using image data captured by the camera array.Furthermore, the above discussion of color channels that do not containa reference camera is presented primarily with reference to red and bluecolor channels. Any color channel, however, can be evaluated usingprocesses similar to those outlined above as appropriate to therequirements of specific applications in accordance with embodiments ofthe invention. Indeed, all color channels captured by a camera array canbe evaluated using processes similar to those described above withreference to FIG. 5E when a super-resolution image is synthesized from avirtual viewpoint (i.e. none of the color channels include a camera thatcaptures image data from the viewpoint from which the super-resolutionimage is synthesized).

Processes for Detecting Defective Camera Arrays

Processes for manufacturing camera arrays, including (but not limitedto) camera arrays implemented using an array camera module, canincorporate processes that screen for defective camera arrays. In manyembodiments, the screening processes identify defects and the regionswithin the cameras in the camera array that capture image data, whichare impacted by the identified defects. The process can then count thenumber of regions impacted by defects within specific sets of regions,where each set of regions constitutes the parallax uncertainty zones fora specific region of a super-resolution image that can be synthesizedusing image data captured by the camera array. In many embodiments, thespecific sets of regions can include different sets for each colorchannel used to synthesize a region of a super-resolution image. In thisway, predetermined parallax uncertainty zones can effectively be definedas a set of look up tables (or similar data structures) without the needto continuously perform the calculations to determine the parallaxuncertainty zones (which are typically the same for each similar cameraarray being manufactured and tested).

A process for screening camera arrays to identify defective cameraarrays in accordance with an embodiment of the invention is illustratedin FIG. 6. The process 600 includes capturing (602) image data of aknown target using multiple focal planes. In many embodiments, thetarget includes features that enable evaluation of captured image datafor the purpose of detecting localized defects within the array camera.In a number of embodiments, a target is used that enables localmeasurement of MTF at multiple field locations such as (but not limitedto) targets that incorporate slanted edge targets (for both tangentialand sagittal components), bar targets (for both tangential and sagittalcomponents) and/or Siemens stars. In certain embodiments, the specifictypes of targets are repeatedly arranged to be imaged into differentregions. The captured image data is divided (604) into regions and anylocalized defects are identified (606) within the regions. Processes inaccordance with embodiments of the invention can screen for multipledifferent types of defects including (but not limited to) defects in thelens stack of a camera, defects in a camera's sensor, and defectsresulting from the incorrect assembly of the camera optics and sensor.As is discussed further below, the process of dividing the capturedimage data into regions can involve dividing the captured image datainto different sized regions for the purpose of evaluating the impact ofdifferent types of images on the image quality of super-resolutionimages synthesized by the camera array.

In a number of embodiments, a reference camera is selected (608). As isdiscussed further below, processes in accordance with many embodimentsof the invention require that the reference camera utilized in thesynthesis of super-resolution images be free of localized defects.Accordingly, the process of selecting a reference camera can involveselecting candidate reference cameras and evaluating whether any of thecandidate reference cameras are free from defects. In the event thatnone of the candidate reference cameras are free from defects, thecamera array may be rejected.

The process of screening the camera array can then involve identifying(610) defects that impact image data captured by regions within theparallax uncertainty zones of each region of a super-resolution imagethat can be synthesized using image data captured by the camera array.As noted above, this can involve utilizing look up tables (or similarrapidly accessible data structures) to count the number of defects thatoccur in specific sets of regions corresponding to the parallaxuncertainty zones (in each color channel) for each region of asuper-resolution image that can be synthesized using the camera array.The number of defects in each of the specific sets of regions can thenbe evaluated to determine (612) whether the number exceeds apredetermined threshold. In many embodiments, different thresholds canbe defined for different sets of regions. In several embodiments,different thresholds apply to the sets in each of the different colorchannels supported by the camera array. In embodiments where the numberof defects in each instance is sufficiently low to satisfy thethresholds, then the camera array is determined to be capable ofsynthesizing super-resolution images of acceptable image quality andinformation concerning the defects can be stored for use by the cameraarray in the subsequent synthesis of super-resolution images. In thisway, information concerning defects can be utilized to disregard imagedata captured by regions of cameras impacted by the defects during thesynthesis of super-resolution images. Processes for synthesizingsuper-resolution images in this manner are discussed further below. Inthe event that at least one of the defect counts with respect to aspecific set of regions exceeds the predetermined threshold, then thecamera array is determined to be defective for the purpose ofsynthesizing super-resolution images having acceptable image quality.

Although specific process for determining whether a camera array isdefective are described above with respect to FIG. 6, any of a varietyof processes can be utilized in accordance with embodiments of theinvention including processes that define sets of regions based upon anyof a variety of criterion appropriate to a specific applicationincluding sets that evaluate the reliability of image data used tosynthesize other types of images including (but not limited to) stereopairs of super-resolution images, sequences of images synthesized fromsub-arrays of cameras within of the camera array, and/or high speedvideo sequences including successive frames synthesized from differentsub-arrays of cameras within the camera array. As can be readilyappreciated in view of the above discussion, the specific regionsincluded within the specific sets of regions can be determined basedupon the cameras used to synthesize each type of image and usingepipolar lines and maximum parallax shift bounds to identify the regionsin those cameras that fall within parallax uncertainty zones. Processesfor identifying defects in accordance with embodiments of the inventionare discussed further below.

Identifying Defective Regions Based Upon Pixel Defects

A region of a camera can be considered defective due the presence ofdefective pixels within the region. Pixels can be considered defectivefor reasons including (but not limited to) the pixels being determinedto be hot pixels, bright pixels, or dark pixels. Any of a variety ofcriteria appropriate to the requirements of specific applications can beutilized to determine whether the presence of defective pixels within aregion renders the entire region defective for the purpose of evaluatingthe camera array. In a number of embodiments, the presence of apredetermined number of pixels results in the entire region beingconsidered defective. In several embodiments, the presence of a clusterof defective pixels exceeding a predetermined size within region resultsin the entire region being considered defective. In certain embodiments,clusters of pixels that are equal to or smaller than a 2×2 cluster ofpixels can be tolerated. However, a cluster of pixels that includesthree or more pixels in one dimension results in the entire region beingconsidered defective. In other embodiments, the size of defective pixelclusters that can be tolerated is determined based upon the requirementsof specific applications.

A process for determining whether the presence of defective pixelsresults in a region being considered defective for the purpose ofevaluating the performance of a camera array in accordance with anembodiment of the invention is illustrated in FIG. 7. The process 700includes detecting (702) defective pixels using image data capturedwithin a specific region. A determination (704) is made concerningwhether the number of defective pixels exceeds a threshold. If thethreshold is exceeded, then the region is considered to be defective(706) for the purpose of evaluating the camera array. In the event thatthe number of defective pixels does not exceed the predeterminedthreshold, a separate determination (708) is made concerning whether thesize of any clusters of defective pixels exceeds a predeterminethreshold. In the event that one or more clusters are present that theexceed the maximum size criterion, then the region is considered to bedefective (706) for the purpose of evaluating the camera array.Otherwise the region is treated (710) as not being defective despite thepresence of defective pixels. In many embodiments, informationconcerning the defective pixels is stored for use when synthesizingimages using image data captured by the region so that the image datacaptured by the defective pixels can be disregarded.

Although specific processes for determining whether a region of a camerais defective based upon the characteristics of defective pixels presentwithin the region of the camera are described above, any of a variety ofprocesses utilizing any of a variety of criteria appropriate to therequirements of specific applications can be utilized to determinewhether a region of a camera is defective for the purpose of evaluatinga camera array based upon the number, type, and/or location of defectivepixels within the region in accordance with embodiments of theinvention. Processes for evaluating whether a region of a camera isdefective utilizing MTF measurements in accordance with embodiments ofthe invention are discussed below.

Identifying Defective Regions Using MTF Measurements

Defects in the optics of a camera can be identified by performing MTFmeasurements. In several embodiments, defects in regions of a camerathat are attributable to defects in the lens stack of the camera can bedetected by performing an MTF measurement for the region. Where the MTFmeasurement diverges from the anticipated MTF of the optics, then MTFfailure can be considered to have occurred within the region and theregion can be treated as defective for the purpose of evaluating theoverall reliability of the camera array.

A process for determining whether a region of a camera is defective whenevaluating the overall reliability of a camera array in accordance withan embodiment of the invention is illustrated in FIG. 8. The process 800includes measuring (802) the MTF of a region of an image captured by thecamera. When a determination (804) is made that the MTF measurementindicates that the MTF within the region falls below a predeterminedthreshold. In many embodiments, when an MTF measurement with respect toa region does not meet a threshold for a certain contrast at a certainspatial frequency, the region of the camera is determined to bedefective for the purpose of evaluating the overall performance of thecamera array. In the event that the MTF measurement for the regionsatisfies the predetermined acceptance criterion, then the region isdetermined (808) not to be defective for the purpose of evaluating theoverall performance of the camera array.

Although specific processes are described above with reference to FIG.8, any of a variety of processes can be utilized to screen regions ofcameras based upon the characteristics of the optics of a camera asappropriate to the requirements of specific applications in accordancewith embodiments of the invention.

Processes for Selecting a Reference Camera

In many embodiments, the process of synthesizing a super-resolutionimage involves selection of a reference camera and synthesizing thesuper-resolution image from the viewpoint of the reference camera. Thecamera selected as the reference camera plays an important role in thesynthesis of the super-resolution image. Therefore, processes inaccordance with a number of embodiments of the invention attempt toselect a reference camera that is free from defects and will discard acamera array when none of the cameras that can serve as a referencecamera are free of defects.

A process for selecting a reference camera in a camera array inaccordance with an embodiment of the invention is illustrated in FIG. 9.The process 900 includes selecting (902) an initial reference camera. Adetermination (904) is made concerning whether the selected camera isfree from defects. In the event that the selected camera is free ofdefects, then the camera is selected (906) as the reference camera. Inthe event that the selected camera incorporates one or more defectiveregions, then the process of selecting (902) candidate reference camerasand evaluating (904) the candidate reference cameras repeats untileither a candidate reference camera is found that is free from defectsand selected (906) as the reference camera or all potential candidatesare exhausted. In which case, the camera array is rejected (910) asdefective. Depending upon the construction of the camera array and therequirements of a specific application, there is typically only a subsetof cameras in the camera array that can serve as a reference camera.

Although specific processes for selecting a reference camera arediscussed above with reference to FIG. 9, any of a variety of processesappropriate to the requirements of specific applications can be utilizedin accordance with an embodiment of the invention.

Screening Optic Arrays

While much of the discussion above has focused on systems and methodsfor screening camera arrays for defects that will prevent the synthesisof images having acceptable image quality, similar techniques can beutilized to screen optic arrays manufactured for use in array cameramodules in accordance with embodiments of the invention. The MTF inmultiple regions of each of the lens stacks in an optic array can bemeasured using an optical test instrument designed to perform MTFtesting, such as (but no limited to) the Image Master® PRO line ofproducts manufactured by Trioptocis GmbH of Wedel, Germany. Furthermore,scripts can be executed on such optical test instruments to detectdefective optic arrays using processes that consider the impact thatdefective regions within the optical array would have on imagessynthesized using image data captured by a hypothetical array camerasthat incorporated the optic array in accordance with embodiments of theinvention. Defects in a lens stack can be localized by separatelymeasuring the MTF of each of a number of regions of each lens stack.Parallax uncertainty zones can be defined with respect to the regions ofthe lens stacks in the optic array in the same way in which they aredefined for regions of cameras in a camera array. By counting regions inthe parallax uncertainty zones that have MTF measurements that fail tosatisfy one or more predetermined MTF criterion, a determination can bemade concerning whether the defects in the optics are likely to resultin the construction of an array camera module that is incapable ofcapturing image data from which super-resolution images can besynthesized with acceptable image quality. As with camera arrays, thespecific set of regions that forms each parallax uncertainty zone can bestored in a lookup table (or similar data structure) enable rapidretrieval. In this way, counts can be generated and the appropriatethreshold applied with respect to each set to determine whether theoptic array is defective.

A process for determining whether a lens array is defective inaccordance with an embodiment of the invention is illustrated in FIG.10. The process 1000 includes measuring (1002) MTFs for differentregions of each lens stack in the lens stack array. Localized defectscan be identified (1004) by comparing the MTF measurements to at leastone predetermined criterion such as (but not limited to) any of thefollowing threshols: on-axis MTF at 227 lp/mm>0.3; all regions at 0.6relative field height having S-MTF at 227 lp/mm>0.2, and T-MTF at 227lp/mm>0.2; all regions @0.8 relative field height having S-MTF at 227lp/mm>0.15, and T-MTF at 227 lp/mm>0.1. In other embodiments, anythresholds appropriate to the requirements of specific applications canbe utilized. In a number of embodiments, a lens stack is selected as areference lens stack. As is discussed above, several embodiments of theinvention require that a reference camera be free from defects.Accordingly, a lens stack selected as a reference lens stack can also besubject to a requirement that it be free from defects. In the event nolens stack that can serve as the lens stack of a reference camera isfree from defects, then certain embodiments of the invention involverejecting the lens stack as defective.

The process of screening the optic array can then involve identifying(1008) defects that will impact image data captured within parallaxuncertainty zones of each region of a super-resolution image that can besynthesized from the viewpoint of the reference lens stack. As notedabove, this can involve utilizing look up tables (or similar rapidlyaccessible data structures) to count the number of defects that occur inspecific sets of regions corresponding to the parallax uncertainty zones(in each color channel) for each region of the reference lens stack. Thenumber of defects in each of the specific sets of regions can then beevaluated to determine (1010) whether the number exceeds a predeterminedthreshold. In many embodiments, different thresholds can be defined fordifferent sets. In several embodiments, different thresholds apply tothe sets in each of the different color channels that will ultimately beformed using the optic array. In embodiments where the number of defectsin each instance is sufficiently low to satisfy the thresholds, then theoptic array is determined to be suitable for use in the construction ofan array camera module. Furthermore, information concerning defectscaptured during the screening process can be subsequently utilized todisregard image data captured by regions of cameras impacted by thedefects in the optic array during the synthesis of super-resolutionimages. Processes for synthesizing super-resolution images in thismanner are discussed further below. In the event that at least one ofthe defect counts with respect to a specific set of regions exceeds thepredetermined threshold, then the optic array is determined to bedefective for the purpose of constructing an array camera module.

Although specific processes for determining whether an optic array isdefective are described above with respect to FIG. 10, any of a varietyof processes can be utilized in accordance with embodiments of theinvention including processes that define sets of regions based upon anyof a variety of criterion appropriate to a specific applicationincluding sets that evaluate the reliability of optic arrays based uponsynthesizing other types of images including (but not limited to) stereopairs of super-resolution images, sequences of images synthesized fromsub-arrays of cameras within of the camera array, and/or high speedvideo sequences including successive frames synthesized from differentsub-arrays of cameras within the camera array. Furthermore, materialbinning can be utilized to further improve yield by combining opticarrays and sensors based upon the defects present in each component. Inthis way, combinations can be created that match regions in whichlocalized defects are present in an optic array with localized defectsin a sensor to minimize the total number of camera regions that containlocalized defects in an array camera module assembled using the opticarray and the sensor.

Synthesizing Images Using Camera Arrays Incorporating Defects

Camera arrays that are screened utilizing processes similar to thoseoutlined above and/or that include optic arrays and/or sensors that arescreened utilizing processes similar to those outlined above can containdefects. When image data captured by pixels impacted by the defects isutilized to synthesize an image, then a degradation in image quality canresult. The process of screening the camera array and/or the opticsyields information concerning regions in the cameras and/or lens stacksor sensors containing defects. In several embodiments of the invention,information concerning the defective regions is maintained by the cameraarray and utilized in the processing of captured image data tosynthesize images. In several embodiments, image data captured in adefective region can be disregarded. Where a region is identified asdefective, but the location of the specific pixels impacted by thedefect is known, only the impacted pixels can be disregarded.

A process for synthesizing a super-resolution image that involvesdisregarding image data captured by regions and/or pixels impacted bydefects in accordance with an embodiment of the invention is illustratedin FIG. 11. The process 1100 includes capturing (1102) image data usingthe cameras in the camera array. Information concerning defectiveregions in specific cameras and or lens stacks, which can take the formof arbitrary formatted defect data, can be utilized to disregard (1104)image data captured by pixels in the impacted regions of the identifiedcameras. It is worth noting that when an entire region is disregarded,even the pixels within the region that are not impacted by the defectare disregarded. In many embodiments, the defect data can also containinformation concerning defective pixels can also be disregarded (1106).A super-resolution process can be applied (1108) to the remaining imagedata and yield a super-resolution image (1110) as an output.

Although specific processes for synthesizing super-resolution images arediscussed above with respect to FIG. 11, any of a variety of processesfor synthesizing images from image data captured by camera arrays thatutilize information concerning regions of specific cameras within thecamera array that contain defects can be utilized in accordance withembodiments of the invention including (but not limited to) processesthat involve synthesizing stereo pairs of super-resolution images,sequences of images synthesized from sub-arrays of cameras within of thecamera array, and/or high speed video sequences including successiveframes synthesized from different sub-arrays of cameras within thecamera array.

While the above description contains many specific embodiments of theinvention, these should not be construed as limitations on the scope ofthe invention, but rather as an example of one embodiment thereof. It istherefore to be understood that the present invention may be practicedotherwise than specifically described, without departing from the scopeand spirit of the present invention. Thus, embodiments of the presentinvention should be considered in all respects as illustrative and notrestrictive.

What is claimed is:
 1. A method for synthesizing a super-resolutionimage using a camera module that includes a plurality of localizeddefects, comprising: capturing image data using a camera arraycomprising a plurality of cameras, where a first camera of the pluralityof cameras includes a known localized defect impacting image datacaptured by the first camera; disregarding image data within a region ofa first image captured by the first camera using a processor configuredby a super-resolution image processing application, where thedisregarded image data comprises image data impacted by said knownlocalized defect of the first camera; and synthesizing asuper-resolution image using a super-resolution process performed by theprocessor configured using the super-resolution image processingapplication, where, for a region of the super-resolution image thatcorresponds to the region of the first image with the disregarded imagedata, image data from a corresponding region of a second image capturedby a second camera of the plurality of cameras is used for thesuper-resolution process, where the corresponding region of the secondimage corresponds to the region of the first image and is identified bysearching for correspondence along an epipolar line up to apredetermined maximum parallax shift distance, where the epipolar lineis defined parallel to the relative locations of the center of the firstcamera and the center of the second camera.
 2. The method of claim 1,wherein the known localized defect of the first camera comprises atleast one defective pixel, and disregarding the image data comprisesdisregarding image data captured by the at least one defective pixel ofthe first camera that is known to be defective.
 3. An array camera,comprising: an array camera module comprising a plurality of camerasformed by an imager array comprising a plurality of focal planes and anoptic array comprising a plurality of lens stacks, where a first cameraof the plurality of cameras formed by the imager array and optic arrayincludes a known localized defect impacting image data captured by thefirst camera; a processor; and memory containing a super-resolutionimage processing application and defect data that identifies the firstcamera that includes the known localized defect and a region of thefirst camera that contains the known localized defect; wherein thesuper-resolution processing application configures the processor to:capture image data using the array camera module; disregard image datawithin at least one region of an image captured by the first camera,where the at least one region of the image comprises image data impactedby the region of the first camera that contains the known localizeddefect as identified by the defect data; and synthesize asuper-resolution image, where, for a region of the super-resolutionimage that corresponds to the region of the first image with thedisregarded image data, image data from a corresponding region of asecond image captured by a second camera of the plurality of cameras isused for the synthesis, where the corresponding region corresponds tothe region of the first image and is identified by searching forcorrespondence along an epipolar line up to a predetermined maximumparallax shift distance, where the epipolar line is defined parallel tothe relative locations of the center of the first camera and the centerof the second camera.
 4. The array camera of claim 3, wherein the defectdata further identifies at least one defective pixel within the imagerarray and the super-resolution processing application configures theprocessor to also disregard image data captured by the at least onepixel identified as defective by said defect data.